
SLDS Spotlight 

Development and Use of Early Warning Systems 


An early warning system is a data-based tool that uses indicators to help predict which students 
are on the right path towards eventual graduation or other grade-appropriate goals. Through 
early warning systems ', stakeholders at the school, district, and state levels can view data from a 
wide range of perspectives and gain a deeper understanding of student data. 

This Statewide Tongitudinal Data Systems (SLDS) Spotlight takes a look at some of the 
early warning system development and expansion work going on in Massachusetts, Maine, and 
South Carolina. It discusses some of the benefits of using an early warning system and offers 
tips on the development of this type of system. 


Massachusetts: Early Warnings for Local Use 

Currently, the Massachusetts Department of Elementary and Secondary Education uses 
an Early Warning Indicator Index (EWII) — a system that was implemented in 2008 to 
target urban districts. In 2011, it was made available to all districts. 

The EWII is used as a starting point to identify and support students who may be at risk 
of not graduating on time (in four years or less) or dropping out of high school. EWII 
assigns all first- time grade 9 students to one of five risk levels (Very High Risk, High Risk, 
Borderline, At Risk, or Low Risk) based on five indicators. For the 2011—12 cohort of 
first-time ninth graders, these indicators are as follows: 

• spring 2011 grade 8 Massachusetts Comprehensive Assessment System (MCAS) 
results; 

• spring 2011 grade 8 English Language Arts (ELA) scores; 

• 2010— 11 attendance rate; 

• number of in- and out-of-school suspensions in the 2009—10 and 2010—11 school 
years; and 

• age as of September 1, 2011. 

Although this system has been refined over the years, it was limited in scope only focusing 
on rising ninth graders. Massachusetts received positive feedback on the information 
provided thru EWII, and many districts were interested in expanding the early warning 
system to provide learning-indicator data from earlier grade levels. After receiving an 
SLDS grant in 2010 (FY09 ARRA), Massachusetts began developing the Early Warning 
Indicator System (EWIS) to focus on P-12 data. Ideally, future versions of the model will 
expand even further to include children from birth through college. 

Unlike the EWII, the EWIS will provide data to districts and other stakeholders through 
web-based technology solutions. Prior to this, the EWII provided Microsoft® Excel files 
to districts through a secure portal. 
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Rather than predicting the likelihood of students’ graduation 
from high school, this model will focus on more proximate 
outcomes, such as the likelihood of reaching proficiency in 
grade 3 reading and passing all grade 9 courses. 

Importance of Stakeholder Input 

Throughout the process of creating a new early warning 
system, Massachusetts relied heavily on stakeholder input from 
both internal and external sources. According to Jenny Curtin, 
Coordinator of High School Graduation Initiatives from the 
Massachusetts Department of Elementary and Secondary 
Education, stakeholders provided critical and objective 
feedback that helped the project group improve and refine the 
system. 

“We learned a lot from our stakeholders at both the state level 
and across districts,” said Ms. Curtin. “Many times, they saw 
something that we may have missed, which produced a much 
better product in the end.” 

Massachusetts also created an advisory group at the state level 
that informs the development of this model on a regular basis. 
The group is comprised of a diverse group of stakeholders, 
including SEA members from education services, education 
data, research evaluation, as well as members from schools 
districts. Along with the advisory group, focus groups and 
interviews will continue through fall 2012. 

Training on the Use of Early Warning Data 

The next step for implementing the EWIS is training that will 
assist districts with the utilization of the data. Notably, this 
training will provide best practices for the use of the data to 
inform interventions. Because Massachusetts had not addressed 
training with the EWII model (development for the new system 
was already under way), the training for EWIS remains a crucial 
component. 

Along with training district staff and other stakeholders on best 
practices, Ms. Curtin hopes to use training sessions to overcome the 
challenges of using early warning systems. For example, Ms. Curtin 
hopes to show that early warning indicator data should be seen as a 
tool for guiding further investigation of student needs rather than 
the interpreted as the reason students are at-risk. 


Maine: Early Warnings at the Classroom Level 


Maine’s early warning system, the Ninth Grade Foundational 


Dropout Model, was designed as a high school dropout early 
warning and management system. With the system, students 
are initially evaluated as they enter grade 9 using research-based 
indicators. After Maine was awarded an SLDS grant in 2006 
(FY07), the state was able to implement a K12 Integrated Data 
System. Like Massachusetts, Maine also received feedback 
from stakeholders who wished to see earlier data that tracked a 
student’s education from early years, and it was clear that a look 
at earlier indicators was needed. 

“By the time you identify the students in grade 9 [who are at 
risk of dropping out], it’s too late to take a lot of action,” said 
Bill Hurwitch of Maine’s Department of Education. 

Maine is currently revising its student data collections for fall 
2012 to add more data elements that will ultimately expand 
the model to include early indicator data. Maine researchers 
will also validate the indicators used for grade 9 are valid for 
elementary and middle school students. Unlike high school- 
level data, grade 3 through 8 data will include annual assessment 
data. 

At-Risk Data Mart 

Maine created the At-Risk Data Mart in 2010 to help educators 
identify students at risk of dropping out. This tool also allows 
educators to create, assign, and manage programs for at-risk 
students; track student performance in the programs; provide 
analysis of model results to help the state to improve the 
model’s accuracy. This model measures academic performance, 
educational engagement, and student background. 

Unlike typical early warning models, Maine’s At-Risk Data 
Mart is specifically aimed at the classroom and state levels. This 


Figure 1. Maine's At-Risk Data Mart 



tool contains two areas of reporting: a Model Roster section, 
which is a quick link with access to all students’ reports, and 
Report sections, which contain data snapshots and data tables 
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for deeper analysis at the district level. The At-Risk Data Mart 
also has a Soft Influences feature that allows educators to enter 
qualitative data that cannot be academically measured, such as 
their observations of and conversations with a student. This tool 
is used by teachers to track the effectiveness of interventions. 

Increase College Readiness 

The At-Risk Data Mart is being leveraged by the Maine 
Community College System (MCCS) Presidents Council to 
help identify high school students at risk of needing remedial 
or developmental courses. Rather than continuing to have 
students wait until college to take remedial or developmental 
courses, the MCCS Presidents Council pushed to deliver 
these courses during a student’s senior year of high school. 
This saves students from paying for a non-credit course and 
provides feedback to high schools regarding which programs 
are (and are not) working. These courses may also influence a 
student’s choice when considering postsecondary education. 

Importance of Validating Measures 

Through building and developing Maine’s early warning system, 
Mr. Hurwitch noted the importance of validating early warning 
system measures and weights with qualified research to fine tune 
the system to improve predictive accuracy. “Measures” are data 
points for a model, while the “indicators” are the groups of 
measures that make up the model. Measures and indicators can be 
weighted to increase their value as part of the overall algorithm. 

In order to validate the measures of the early warning system, 
Maine first looked at historical data and applied indicators to 
past students to see how well the system was able to predict 
individual student dropouts. According to Mr. Hurwitch, 
applying various combinations of historical data helps 
determine the reliability in predicting outcomes. 

A Look to the Future 

The At-Risk Data Mart was developed in 2010 and testing started 
in 2011. Through the work of researchers, Maine recognized 
the need to collect additional data from local education agencies 
(LEAs) before fully re-rolling out the At-Risk Data Mart. Maine 
is working to re-release the dropout model and the subsequent 
college-readiness model by fall 2012. 


South Carolina: Capturing Every Student 


The Student Potential Performance Snapshot (SPPS) was first 


developed as an output of the South Carolina Education and 
Economic Development Act (EEDA), which coincided with an 
SLDS grant in 2005 (FY06). SPPS is a data reporting solution 
designed to inform educators of the at-risk characteristics evident 
in their student population. This early warning system provides the 
information needed for educators to select early and appropriate 
interventions for students who may be at risk of not advancing or 
graduating. The initial research, design, and data points of SPPS 
were formed by the EEDA At-Risk Student Committee. 

The SPPS collects data for all students. Data for SPPS are 
stored in the South Carolina longitudinal data warehouse, 
which includes many years of longitudinal student data — dating 
back as far as grade 3 for some current high school students. 
The primary users of this solution are school personnel 
responsible for counseling students on grade advancement 
and/or graduation. School administrators responsible for at- 
risk models may also use SPPS. 

There are a number of summary reports that show snapshot 
views of the entire student population at different levels of 
data: state, region, district, and school. The state-level report 
includes the total students for the state and totals by at-risk 
indicators for a single year, as well as aggregated information 
for each region, district, and school by student and at-risk 
indicator for a given year. The school summary report includes 
data on each student in the selected school for the current 
school year. The Student Detail Report, which is also referred 
to as the Student Journey, includes data on a student’s entire 
“journey” through the South Carolina public school system. 

The A-B-C’s of Dropout Prevention 

There are 27 indicators used in the SPPS that make up the Student 
Detail Report/Student Journey and fall under the following 
categories: socioeconomic; credits earned; assessment; student 
enrollment journey; discipline, expulsion, and attendance; retention 
and grade; and at-risk indicator programs (used to identify any 
prevention programs designated for that student). 

One of the subcomponents of the SPPS system is the At-Risk 
Index that is made up of 10 of the 27 indicators. These 10 indicators 
include: daily absences, overage indicator, credits earned, suspensions, 
latest score of Priority Academic Student Skills (PASS) English 
Language Arts (ELA) and PASS Math, math grade, discipline events, 
and English proficiency. These indicators are assigned a weight based 
on criteria that are evaluated each time the Student Detail Report is 
executed. The weighted values for the 10 indicators are then added 
together to determine the At-Risk Index Score. 

Of the 27 indicators, three of the indicators are considered the most 
important: attendance, behavior, and credits earned. Based on 
research by Johns Hopkins University and the National Dropout 
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Prevention Center, 1 * these three indicators, also known as the ‘A-B- 
C’s of dropout prevention,” should be tracked for students who are 
flagged as at-risk. The A-B-C indicators also comprise part of South 
Carolina’s At-Risk Index. 

In addition to creating data reports, South Carolina uses Microsoft® 
Excel and Excel PowerPivot, which lets users manipulate and graphically 
depict large data sets. South Carolina’s At-Risk Index makes it easier for 
administrators and counselors to view and interpret the data. 

Merging Data to Expand Knowledge 

Another unique feature relatively new to the SPPS is the 
merging of South Carolina’s data with other data sources for 
use in graphic displays. These visualizations have been made 
with other data that focus on the dropout rate of students. The 
SPPS takes the student potential snapshot and matches it against 
another data source that is used for drop out reporting. Merging 
two data sources that display similar information helps districts 
and other stakeholders see the data from multiple perspectives. 

In Figure 2, the data for the charts and graphs come from the 
summary tables in South Carolina’s data warehouse. The data shown 
in this worksheet screenshot are derived from quarterly district data 
collections. The tabs along the bottom present different views 
of the data. On the left are pivot tables of the snapshot, which 
can be used to customize the graphics by selecting specific school 
years. According to Figure 2, data related to students who dropped 
out during previous school years combined with the current 
SPPS indicators reveal that 63.6 percent of those dropouts had a 
number of discipline incidents, which increased their probability of 
dropping out. Having access to this type of information will help 
educators determine whether existing policies need to be reviewed 
or new policies should be established. 

1 National High School Center (2008). Developing Early Warning 

Systems to Identify Potential High School Dropouts. 


As South Carolina’s longitudinal data warehouse expands, the 
Student Journey will also be able to include data on a student’s 
career from Pre-K through grade 12. These indicators are 
eventually entered into the South Carolina data warehouse. 
Currently, data are available primarily for students in grades 
8—12. The state also plans to include additional assessment 

Figure 2. Sample screenshot of South Carolina's SPPS 
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data (e.g., MAP, end-of-course, and high school exit exam). 
Although South Carolina is currently concentrating on risk 
indicators for K12 data, the state is also working to add 
composite indicators for early childhood education. Although 
these indicators are considered sensitive and may not be 
displayed separately, indicators relevant to a student’s academic 
career will be included for that student’s Detail Report. 

The ultimate goal is to develop a system that will allow 
educators to identify and arrange appropriate interventions for 
potential dropouts as soon as possible. 


State Tips for Developing and Using Early Warning Systems 

• Rely on stakeholder (internal and external) input throughout the development process to improve and refine the 
state’s early warning system. 

• Create a plan to regularly revisit and refine risk models. 

• Validate early warning system measures and weights by conducting quality research. 

• Train stakeholders on best practices regarding early warning systems. 

• Use visuals to help stakeholders view data from multiple perspectives. 
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